Systems and methods for an augmented reality platform

ABSTRACT

Systems, methods, and media for augmenting a view of reality. In an embodiment, a medium, a first marker, and metadata are received from a depository. The first marker comprises at least a portion of a view of reality, and is matched to at least a first portion of the view of reality. Subsequently, a second marker is generated based on at least a second portion of the view of reality, and uploaded to the depository. In addition, the medium is superimposed over at least a portion of the view of reality, based on at least one of the first marker and the second marker, to generate an augmented view of reality.

PRIORITY

This application is a continuation of U.S. patent application Ser. No.13/549,081, filed on Jul. 13, 2012 and titled “Systems and Methods foran Augmented Reality Platform,” which claims priority to U.S.Provisional Patent App. No. 61/508,500, filed on Jul. 15, 2011 andtitled “Systems and Methods for an Augmented Reality Platform,” theentireties of both of which are hereby incorporated herein by reference.

FIELD OF THE INVENTION

This application relates to an augmented reality platform utilized forstorage, retrieval and viewing of geo-located media related to a view ofreality.

BACKGROUND

Augmented Reality generally refers to altering a view of reality.Artificial information about a real environment can be overlaid over aview of the real environment. The artificial information can beinteractive or otherwise manipulable, providing the user of suchinformation with an altered, and often enhanced, perception of reality.However, Augmented Reality is still a relatively new area of interestwith limited present-day applications. For instance, the first-down lineoften seen in television broadcasts of American football games is oneexample of a limited application of Augmented Reality.

The iPhone®, iPad®, more recent Android-platform smart phones, andAndroid tablets offer an array of technologies not seen before in mobilehandheld devices. The high bandwidth connections, Global PositioningSystems (GPS), and high-speed processors and positional sensors offernew opportunities in the field of Augmented Reality.

SUMMARY

As disclosed herein, this technology can be leveraged into a moreimmersive and communal experience of Augmented Reality on a smart phoneplatform. Imagine a virtual gallery where photos, history, andfuturistic designs are viewable from the original artist's vantagepoint, fixed in free space, while superimposed over reality. Furtherimagine that this virtual gallery can be created and shared by friends,family, or any other social group. Imagine being able to take a pictureor video of yourself or your friends interacting with the virtualgallery. The result is a unique social experience of the world, wherethe notion of feeling like you were there will be supplanted by actuallybeing there.

The unseen, the unnoticed and the unimaginable will be available simplyby linking with new friends or social groups and accessing their virtualgalleries. Old discarded photos may receive new life by beingincorporated into an interactively augmented version of reality. Socialgroups can strengthen their connections by experiencing the past,present, and future in the same manner. Normal experiences, such awalking or sight-seeing, can be augmented and enhanced at every turn,rendering daily routines into something amazing and communal, andintimately connecting individuals with their surroundings as seenthrough multiple eyes and over numerous time periods. Indeed, usersexperiencing their surroundings, augmented by past or future media, mayexperience something akin to time travel. These experiences and more canbe enabled and realized by one or more embodiments of the AugmentedReality Platform disclosed herein.

In one embodiment, a system for generating an augmented reality isprovided. The system comprises a virtual map module configured toreceive a medium, receive a marker associated with the medium, receivemetadata related to at least one of the medium and the marker, and storethe medium, the marker, and the metadata in the at least one memory. Inan embodiment, the marker may represent at least a portion of a view ofreality which is related to the medium. The virtual map module may befurther configured to receive a request for the medium, retrieve themedium, the marker, and the metadata from the at least one memory inresponse to the request for the medium, and send the medium, the marker,and the metadata in response to the request for the medium.

In a further embodiment, the medium, the marker, and the metadata arereceived by a client device in communication with the virtual mapmodule. A current camera view of the client device can then be comparedto the marker. When a match between the marker and the current cameraview is confirmed, the current camera view can then serve as the markerfor purposes of superimposing the medium. In other words, the markerreceived from the virtual map module is switched for the current cameraview. Accordingly, the current camera view, instead of the receivedmarker, is used for superimposition of the medium.

In an embodiment, a system for generating an augmented view of realityis provided. The system comprises a client perspective module stored ona client device which is configured to superimpose a first medium over afirst view of reality, receive one or more of a change in transparencyof the superimposed first medium, a change in size of the superimposedfirst medium, and a change in position of the superimposed first medium,generate a first marker, generate first metadata related to at least oneof the first medium and the first marker, and send the first medium, thefirst marker and the first metadata to a depository. In an embodiment,the first marker comprises at least a portion of the first view ofreality. The system may also comprise a client viewer module stored onthe client device which is configured to retrieve a second medium, asecond marker and second metadata from the depository, match the secondmarker to at least a portion of the second view of reality, andsuperimpose the second medium over the at least a portion of the secondview of reality to generate an augmented view of reality.

In a further embodiment, a method for generating an augmented view ofreality is provided. The method comprises receiving a medium, receivinga marker associated with the medium, receiving metadata related to atleast one of the medium and the marker, and storing the medium. In anembodiment, the marker is a representation of at least a portion of aview of reality which is related to the medium. The method may furthercomprise receiving a request for the medium, retrieving the medium, themarker, and the metadata from the at least one memory in response to therequest for the medium, and sending the medium, the marker and themetadata in response to the request for the medium.

In an additional embodiment, a method for generating an augmented viewof reality is provided. The method comprises superimposing a medium overa view of reality, receiving one or more of a change in transparency ofthe superimposed medium, a change in size of the superimposed medium,and a change in position of the superimposed medium, generating amarker, generating metadata related to at least one of the medium andthe marker, and sending the medium, the marker and the metadata to adepository. In an embodiment, the marker comprises at least a portion ofthe view of reality.

In another embodiment, a method for generating an augmented view ofreality is provided. The method comprises receiving a medium, a markerand metadata from a depository, matching the marker to at least aportion of a view of reality, and superimposing the medium over thematched at least a portion of the view of reality to generate anaugmented view of reality. In an embodiment, the marker comprises atleast a portion of the view of reality.

BRIEF DESCRIPTION OF THE DRAWINGS

The details of one or more embodiments of the present invention, both asto structure and operation, may be gleaned in part by study of theaccompanying drawings, in which like reference numerals refer to likeparts, and in which:

FIG. 1 illustrates a display for obtaining information about a user of aclient device, according to an embodiment;

FIG. 2 illustrates a display for obtaining authentication informationfrom a user of a client device, according to an embodiment;

FIG. 3 illustrates a display by which a user may view and select amedium to associate with a view of reality, according to an embodiment;

FIG. 4 illustrates a display by which a user may input informationassociated with a medium, according to an embodiment;

FIG. 5 illustrates a map display which enables a user of a client deviceto indicate a location associated with a medium, according to anembodiment;

FIG. 6 illustrates a display by which a user can view and select mediastored on the client device, according to an embodiment;

FIG. 7 illustrates a display by which a user may input informationassociated with a medium, according to an embodiment;

FIG. 8 illustrates a view of reality, according to an embodiment;

FIG. 9 illustrates a medium superimposed over a view of reality,according to an embodiment;

FIG. 10 illustrates a medium superimposed over and aligned with a viewof reality, according to an embodiment;

FIG. 11 illustrates an overlay for a view of reality, according to anembodiment;

FIG. 12 illustrates a timer which may be configured for capturing amarker associated with a medium, according to an embodiment;

FIG. 13 illustrates a prompt for a user of a client device to confirmthat the capture of a marker associated with a medium is acceptable,according to an embodiment;

FIG. 14 illustrates a prompt for a user of a client device to uploadcaptured information to a Virtual Map, according to an embodiment;

FIG. 15 illustrates a display of the contents of an Exchangeable ImageFile Format file, according to an embodiment;

FIG. 16 illustrates a display for obtaining information about a user ofa client device, according to an embodiment;

FIG. 17 illustrates a display for obtaining authentication informationfrom a user of a client device, according to an embodiment;

FIG. 18 illustrates a display for listing media available for viewing,according to an embodiment;

FIG. 19 illustrates a carousel display for listing media available forviewing, according to an embodiment;

FIG. 20 illustrates a map display for listing media available forviewing, according to an embodiment;

FIG. 21 illustrates an Augmented Reality Window which displays a view ofreality with superimposed indications of media available for viewing,according to an embodiment;

FIG. 22 illustrates an Augmented Reality Window which displaysinstructions to aid the user of a client device in positioning theclient device, according to an embodiment;

FIG. 23 illustrates an Augmented Reality Window which displays asuperimposed medium over a view of reality, according to an embodiment;

FIG. 24 illustrates an Augmented Reality Window which displays asponsored message, according to an embodiment;

FIG. 25 illustrates an Augmented Reality Window which displaysinformation associated with a medium, according to an embodiment;

FIG. 26 is a high-level block diagram of a process for maintaining aVirtual Map, according to an embodiment;

FIG. 27 is a high-level block diagram of a process for associating amedium with a view of reality, according to an embodiment;

FIG. 28 is a high-level block diagram of a process for augmentingreality, according to an embodiment; and

FIG. 29 illustrates a processing system on which one or more of themodules described herein may reside, according to an embodiment.

DETAILED DESCRIPTION

Systems and methods for generating an augmented reality using anAugmented Reality Platform are provided. After reading this description,it will become apparent to one skilled in the art how to implement theinvention in various alternative embodiments and alternativeapplications. Although various embodiments of the present invention aredescribed herein, it is understood that these embodiments are presentedby way of example only, and not limitation. As such, this description ofvarious alternative embodiments should not be construed to limit thescope or breadth of the present invention. Furthermore, unless otherwisespecified, when references are made herein to an action or step beingperformed, the steps are performed by the Augmented Reality Platformimplemented on a client device or server.

1. Virtual Map

The Virtual Map (VM) is a depository for the media which makes up thevirtual world. In an embodiment, this media may include any type ofimage (e.g., photographs, illustrations, designs, icons, and the like),video recordings, three-dimensional models, animations, audiorecordings, narratives and other descriptions, including words andstatistics, links to websites, Really Simply Syndication (RSS) feeds,and any other type of information that can be related to an environment.Alternatively, the media stored in the VM can be limited to certaintypes of information, such as images only, videos only, images andvideos only, etc. The VM can also act as a depository for userinformation and customer-related data (e.g., authentication information,including usernames and passwords, and other types of accountinformation, such as contact information and the like).

The VM may comprise one or more databases in which the media is stored.Information related to the media (i.e., metadata), as well as userinformation and customer-related data, may also be stored in thedatabase(s) or in a separate database or databases. In an embodiment,the databases are relational databases and retrieval is performed usingStructured Query Language (SQL). In response to retrieval requests, theVM can query the database(s) to retrieve and return media, metadata,user and/or customer information, and/or other information stored in thedatabase(s). In response to save requests, the VM can save or modify themedia, metadata, user and/or customer information, and/or otherinformation in the database(s).

In an embodiment, the VM resides on a server or multiple servers withina network. The server or servers may be accessible via one or morenetworks, which may include the Internet. The VM may implement anapplication programming interface (API). In one embodiment, the API isan Extensible Markup Language (XML) API. In this embodiment, requests tothe VM and responses from the VM are packaged in XML. In an embodiment,sensor data and other information associated with the media can be,partially or wholly, packaged or encoded with the associated medium, forexample, in an Exchangeable Image File Format (EXIFF) file. A personhaving ordinary skill in the art will appreciate that there are a myriadof suitable APIs and communication formats available.

In an embodiment, access to the VM may be restricted. For instance,authentication may be required before the VM will respond to a request.In addition, one or more permissions may be associated with a user ofthe VM. For instance, a user may be restricted to receiving only mediawhich was generated by members of a social group of which the user isalso a member. The VM may operate in conjunction with or in a similarmanner as a social networking application, such as Facebook®. In such anembodiment, an originator—or architect—of a medium may associate themedium with permissions. These permissions may be stored in metadataassociated with the medium, for example, in one or more databases. Thepermissions may restrict access of the medium to specific users orgroups of users. Certain permission(s) or the absence of permissions mayallow public access to the medium.

According to an embodiment, when the originator of a medium uploads orotherwise saves or modifies the medium, a notice can be sent or posted.For example, a notice can be posted to a website or other applicationassociated with the originator, such as a Facebook® Wall. In addition, anotice can be posted on a website or other application associated withone or more members of a social group of which the originator is amember or with which the originator is associated. For example, thenotice may be posted to profiles of users which have permission to viewmedia associated with the originator. In addition or alternatively, anotice, such as an email, text message, application message, or thelike, can be sent to members of the social group or to permitted users.The notice may be sent to a mobile device of the members and/orpermitted users.

The notice may comprise information related to the medium. For example,the notice may comprise a link which enables display of the medium as anaugmentation of reality. The notice may comprise an identity of theoriginator of the medium. In an embodiment, the notice may comprise alocation where the medium was created. The notice may also comprise adescription of the medium or a comment related to the medium. The noticemay further include information, such as directions or a hyperlink,which enables viewers of the notice to download a client applicationassociated with the Augmented Reality Platform.

2. Client Application

The client application enables creation or saving of media for viewingby users of the Augmented Reality Platform, and/or the viewing of themedia as an augmentation of reality. These functions may be embodied ina single client application. Alternatively, the functions can beperformed by separate client applications, such as a client perspectiveapplication and a client viewer application. While a client perspectivemodule and client viewer module are separately referred to herein, aperson having ordinary skill in the art will appreciate that thesemodules may be implemented as a single module or separate modules, andmay be implemented in a single client application or as separate clientapplications.

The client application(s) may reside on a client device, such as a smartphone, tablet device, or other device which includes or is incommunication with one or more cameras or optical viewers. The clientapplication is in communication with the VM (e.g., over one or morenetworks, such as the Internet) and implements the API of the VM. In anembodiment, the API is an XML API, and the client application packagesrequests and receives responses in XML format. However, a person havingordinary skill in the art will appreciate that there are a myriad ofsuitable APIs and communication formats available.

a. Client Perspective Module

In an embodiment, a client perspective module of the clientapplication(s) is configured to receive information from sensorsembedded or otherwise connected to the client device. In an embodiment,the sensors include a GPS, an accelerometer, and/or a compass. Thesensor(s) permit the client perspective module to obtain data related tothe location and orientation of the client device. In an embodiment,this data includes the geographical location (geo-location), yaw, pitch,and roll of the client device. This data may also include a time of dayand/or existing lighting conditions. Geo-location data can comprise thelatitude, longitude and/or elevation of the client device. In additionor alternatively, the geo-location data may comprise a street address orother address associated with the location of the client device. Aperson having ordinary skill in the art will appreciate that othersensors can be utilized and/or other information can be collected andhelpful in determining the location and orientation of a client device.

The client perspective module is configured to receive an image and/orvideo from a camera or other optical viewer. A user of the clientperspective module may generate an image, video, or other medium bypositioning the client device such that a desired image of reality isvisible in the optical viewer, and then indicating that the clientdevice should capture the medium, e.g., by pressing a physical orvirtual (e.g., touch-screen) action button. The client perspectivemodule can capture the medium along with information from the sensor(s)regarding the location and orientation of the device at the time themedium was captured. In this embodiment, the captured medium is also amarker which can be used by the client viewer module forpattern-matching. The client perspective module can also capture thetime of day and/or existing lighting conditions at the time that themarker is captured.

In an embodiment, the client perspective module is configured to enablethe user of the client device to “pin” previously created media, such asan image, video, or other medium, to a view of reality. For example, theuser of the client device can obtain a digitization of a historicalphotograph or other historical medium and store the digital image in amemory on the client device. The user may then return to the locationwhere the photograph was originally taken and utilize the clientapplication to match the photograph to reality.

In at least some of the embodiments disclosed herein, an individual isable to give new life to, for example, a historical photograph. The usermay, first, need to digitize the photograph, e.g., by scanning the imageinto a digital file using a digital scanner. Next, the user canphysically transport the digital file (e.g., on a smart phone or otherdigital device) to the location where the historical photograph wastaken. The user can then “pin” the digitized photograph onto the realworld (e.g., using the client perspective module). Essentially, thehistorical photograph is hung up, like a poster, in the real world.While the photograph is “hung” in a digital form, the digitizedphotograph is not bound to a purely digital world. Rather, friends,family, social groups, etc., can experience and interact with thesubject matter of the digitized historical photograph in the real world(e.g., using the client viewer module). In this manner, the digitalworld and the real world are merged into an augmented reality.

According to one embodiment, the client perspective module cansuperimpose the medium (e.g., image such as a photograph) over a view ofreality visible in the optical viewer of the client device. The clientperspective module can apply a transparency to the medium. The level oftransparency (e.g., percentage of transparency) applied to the mediumcan be a predetermined application setting, or a user setting with orwithout a predetermined default value. In embodiments in which thetransparency is a user setting, the client perspective module mayoverlay a slider bar over the view of reality. The user may theninteract with the slider bar (e.g., by moving a selector from one end ofthe slider bar to the other end of the slider bar) to easily adjust thetransparency of the medium from one end of a spectrum (e.g., 100%, 99%,98%, etc. transparent) to the other end of the spectrum (e.g., 100%,99%, 98%, etc. opaque).

The user can adjust the scale of the medium while it is superimposedover the view of reality in order to match the natural scale of the viewof reality. For example, in embodiments which utilize a touch-screeninterface, the scale of the medium may be adjusted using “pinch” and“spread” interactions. For example, the user can place two fingers(e.g., index finger and thumb) on the medium, and pinch the fingerstogether to reduce the size of the medium or spread the fingers apart toincrease the size of the medium. The user can also orient the clientdevice and/or move the medium so as to align the superimposed mediumwith the view of reality in the optical viewer. For example, inembodiments which utilize a touch-screen interface, the user may touchand drag the medium around a display of the view of reality in order toalign the medium with the view of reality. In these manners, the clientperspective module may be configured to enable the user to adjust thesuperimposed medium, such as by resizing the medium, rotating the mediumin two or three dimensions, or otherwise adjusting the medium within theoptical viewer. This ability creates a methodology for manual inversephotogrammetry, which is the discovery of the original location for thecontent of a medium through superimposition of the medium on reality.Additionally or alternatively, the client perspective module can beconfigured to at least attempt to automatically align the superimposedmedium, for example, by resizing or rotating the medium to match apattern or patterns in the medium to a pattern or patterns in the viewof reality.

A medium is aligned with the view of reality when features of the medium(e.g., landmarks and/or other distinguishing features) are geometricallycongruent with corresponding features of the view of reality. When thephotograph or other medium is in alignment or near alignment with theview of reality, the augmented view can be captured either automaticallyor in response to a user interaction, such as a selection of a physicalor virtual action button. The client perspective module captures theview of reality or a portion of the view of reality and the superimposedmedium's relationship with the view of reality in the optical viewer atthe time of capture, i.e., the manner in which the user or clientperspective module has aligned the medium with the captured view ofreality. In this embodiment, the captured view of reality or portion ofview of reality is the marker which can be used by the client viewermodule for pattern-matching. The client perspective module can alsocapture information regarding orientation of the client device from thesensors. Location information, such as latitude and longitude oraddress, can also be captured from the sensors, or alternatively, inputby the user of the client device. In an embodiment, the medium mayalready have associated location and/or orientation information. Otherinformation may be input by the user, such as date, time, title,description, comment, hyperlink, information related to the user of theclient device, etc.

Both the captured view of reality and the medium (e.g., historicalphotograph), either separately or overlaid/merged, may be encapsulatedor referenced in an embeddable algorithm or code (e.g., HTML code) thatcan be shared via networks, such as the Internet and/or wirelessnetworks for mobile communications on mobile devices andInternet-connected devices. For example, this embeddable code can beplugged into a HTML-based webpage or into a software application. Theembeddable code may allow users to see an individual image or morphedimage of the captured view of reality and medium in memory, as well asadjust the transparency via a manual slide or other interactive input,and view the progression from the view of reality to the morphed image.The code may be written in versatile and flexible multi-platform orcross-platform code (e.g., Java, JavaScript, Flash, etc.) that can beeasily shared and/or embedded. In an embodiment, the code aligns themedium with the captured view of reality for presentation on a pluralityof devices and across one or more networks, including the Internetand/or one or more wireless networks.

In an embodiment, once a medium has been pinned using the clientperspective module, both the captured view of reality and the medium areencapsulated into an embeddable algorithm that can be shared by means ofthe Internet, wireless networks for mobile communications, and/or othernetworks. The embeddable algorithm may utilize a plug-in, and layers themedium and the captured view of reality. The algorithm also appliesimage filters that can be adjusted by the user to create a composite“then and now” image. For maximum compatibility with mobile and networkdevices, the algorithm or plug-in can be written usingplatform-independent and operating system-independent languages and/ortechnologies (e.g., Java).

It should be understood that the medium being pinned need notnecessarily be historical or previously created. For instance, the userof the client perspective module may generate the medium simultaneouslywith the marker. In some cases, the medium may itself be the marker orthe basis for the marker. For example, a user could use the clientperspective module to capture a photograph of street art painted on anexternal wall of a building. The client perspective module can generatea marker from the captured photograph or geometric features of thecaptured photograph. In this case, the medium is generatedsimultaneously or contemporaneously with the marker. Subsequently, aclient viewer module can use the marker and/or orientation informationto align the medium and, for example, display the street art to a uservia a client device, even after the street art has been painted over orotherwise obscured or destroyed.

According to an embodiment, the view of reality is captured by aninfrared or near infrared (IR) receiver. The IR receiver may be embeddedor otherwise in communication with the client device. In thisembodiment, the marker may comprise an encoded thermal image of the viewof reality. This thermal image marker, which may exist in thenon-discernable IR spectrum, enables recognition of the marker, and thusalignment and viewing of the medium, in both daylight and nighttimesettings. Such a thermal image marker may be non-discernable in that itwould not typically be noticed or viewable by the naked eye in realityand may not itself be useful or enjoyable as an image. Thus, in anembodiment, the marker may have no use outside the system. Rather, themarker acts as a trigger for the system. For example, the marker may actas a pattern which is recognized by, and triggers a response (e.g.,superimposition and alignment of a medium to a view of reality) from,the client viewer module. In this manner, the client perspective moduleis able to utilize and appropriate reality's schemes, visual cues,designs, lighting, and other visual information for the purposes of thesystem (e.g., client viewer module).

In addition to this infrared or near infrared marker, the clientperspective module can detect and generate point clouds of physicalsurroundings. The point clouds may comprise a plurality ofthree-dimensional points representing one or more objects in the view ofreality. Each point may comprise x, y, and z coordinates, and anassociated color. The coordinates may be relative to the position of theclient device, relative to the earth (e.g., by translating thecoordinates relative to the client device into GPS coordinates based onthe client device's GPS location), or relative to some other baseline.The point clouds can be stored as metadata with the marker, and can beused by the client viewer module to orientate the marker and reveal themedium when the point cloud is matched to a formation of physicalsurroundings. The software for generating the point clouds by the clientperspective module and matching the point clouds to physicalsurroundings by the client viewer module can be an add-in, for example,that is sold as an upgrade or additional feature of the respectivemodules.

According to an embodiment, after the marker is captured, the clientperspective module may request information from the user, for instance,by displaying inputs with which the user can interact. For example, theclient perspective module may display input text boxes, into which auser can enter information, such as date, time, title, description,comment, hyperlink, and/or information related to the user of the clientdevice. The client perspective module may also display a map on whichthe user can indicate a location to be associated with the medium, forexample, by tapping on a location on the map.

The marker and the medium can be packaged into a save request by theclient perspective module, which is then forwarded by the clientperspective module to the VM for storage and future retrieval andviewing. In the embodiment or scenario where the marker is the medium(e.g., the view of reality is captured as the medium to be hung), thesave request can include two copies of the medium, one designated as themarker and the other designated as the medium, or it may include onlyone copy of the medium. Relationship information between the marker andmedium can also be packaged into the save request. For example, therelationship information may include dimensions, offsets, and/or anglesof rotation of the medium, which are required to align the medium withthe marker. In addition, captured sensor data may also be packaged intothe save request. Other information may be packaged into the saverequest as well, such as a date, time, title, address, description,comment, hyperlink, information related to the user of the clientdevice, etc. In an embodiment, the sensor data, relationshipinformation, and other information can be, partially or wholly, packagedor encoded with the marker and/or medium, for example, in an EXIFF file.

In an embodiment, the media and associated metadata and otherinformation can be subsequently edited or added to by the originatorsand/or other users with access to the media. For example, an originatinguser may want to update a description of the medium, or anon-originating user may want to add a comment about or rating of themedium. In either case, the updated or additional information can bepackaged into a modification or save request and forwarded to the VM forstorage.

In an embodiment, the marker is more than simply an image of the view ofreality. The marker represents a calculated vector from the clientdevice, whereas the image is the visual representation of the calculatedvector. The vector may comprise yaw, pitch, roll, altitude, directionalinformation, and the like. This enables the client perspective module tocapture reality at different vectors and allows the system (e.g., theclient viewer module) to create “effects,” or user experiences,utilizing a plurality of neighboring vectors. As the number of vectorsgrow and develop over time, a connected web of data and images can beutilized to create a fixed, augmented world from multiple views,comprising a multitude of desired media or effects.

In the disclosed embodiments, a client device, such as a mobile orhandheld device, becomes a tool for building an augmented world with amyriad of effects. Such devices enable users to create and fixaugmented-reality media or effects to locations and positions in thereal world. In this manner, the augmented reality is built andpositioned in the field, rather than in a lab. Generation ofaugmented-reality effects in the field and on location permits thecalculation of more accurate vectors than would be possible in the lab.The effects are then fixed in location and position for subsequentviewing, for example, by any user utilizing a client viewer module.

According to an embodiment, once a user of the client perspective modulehas generated a marker corresponding to a certain location and position,the user may generate additional media/effects defined by vectorpositions which are relative to the generated marker. In this manner,multiple media/effects may be associated with a single marker.

The continual addition of vector information to the system allows thesystem to become more accurate, and specific algorithms can beimplemented to reinforce the augmented reality. For example, analgorithm can be used to establish a baseline of positional informationfrom the precision of the client device. If the client device capturingthe marker is accurately tuned to the true global and local position inthree-dimensional space, the marker can confirm any future users'relative position to the original pin-pointed position. Thus, the futureusers' sensor data can be calibrated to reflect the sensor data thatwould be generated by the client device which captured the marker. Thiscalibration establishes a baseline and known delta for a specific user,region, local area, or device. In a similar manner, a specific objecthaving a fixed and known true position can be utilized for calibrationof a user's device. In this way, each client device can calibrateitself, enabling the devices to more accurately position an addedaugmented reality effect at its true position within a visual horizon.

In an embodiment, a three-dimensional marker can be generated. In thisembodiment, a 360° panorama of a horizon or an omni-directional imagecan serve as the three-dimensional marker. It should be understood thatthe three-dimensional marker need not be a full 360° view, but mayinstead comprise some lesser degree, such as 90°, 180°, 270°, etc. Themedium to be pinned can be divided up based on orientation and markerlimitation to recreate a specific effect when the view is confirmedagainst (e.g., pattern-matched to) the three-dimensional marker. Thethree-dimensional marker can be generated (e.g., by the clientperspective module) and matched (e.g., by the client viewer module), forinstance, by aligning the view of reality of the client device to aninitial direction and then rotating the client device with the opticalviewer positioned outwardly to capture a panoramic (e.g., 360°) view ofthe environment. This omni-directional view can be captured by theclient device and used to either generate the three-dimensional markeror match to a previously generated three-dimensional marker.

In an embodiment, omni-directional cameras and high-density photography(e.g., at or approaching giga-pixel levels) can be utilized to create aunique and highly detailed image from a singular point of view. Theelectronic representation of this image, at full resolution, mayrepresent a range that is matched only by film and telescopictechniques. This image can serve as the outermost perimeter or baselinethat is stored in the VM, and can be electronically degraded to recreatelower-resolution lens systems in client devices. Thus, a lowerresolution view can be generated through computation, and the radialvector can be created from the higher, native resolution system thatserves as a marker for the lower-resolution client device. Utilizingthis method, the relative position between the initial stationaryomni-directional high-density image, three-dimensional model, oranimation and the client device can be approximated through inversephotogrammetry and/or visual algorithms which compare a current view ofreality with the high-resolution omni-directional marker stored in theVM. This enables the user to walk through a virtual model in anaugmented reality setting.

In an embodiment, a three-dimensional medium can be pinned to a view ofreality. For example, a user can select a three-dimensional medium(e.g., three-dimensional image). The user can then position thethree-dimensional medium in the view of reality using the clientperspective module. The user could then rotate the client device withthe optical viewer facing inward to capture a three-dimensional view ofreality and generate a three-dimensional marker using the clientperspective module.

In an embodiment, once a marker has been generated, the pinned mediumand any digital effects can be switched out by users at will, and evenfor an indefinite amount of time. Utilizing the generated marker as theparent file, associated effects can be updated, changed, and recreatedas the user sees fit. These modifications may be performed at any laterdate, for example, by accessing a user profile and updating the contentfor the specific marker file.

b. Client Viewer Module

In an embodiment, a client viewer module is configured to allow users ofthe Augmented Reality Platform to view media as an augmentation ofreality. The client viewer module may require a user to log in to anaccount associated with the Augmented Reality Platform or a third-partyservice provider, such as Facebook®. The login may requireauthentication of the user, for example, by requiring the user to entera username or email address and a password associated with the usernameor email address. Following authentication, the client viewer module maythen access the VM. In an alternative embodiment, no account orauthentication is required.

In an embodiment, the user of the client device can search for mediaavailable for viewing. Search fields may include, for example, theidentity of the originator or other user associated with the medium,date and/or time the medium was captured, distance to the medium fromthe current location of the client device or another location, type ofmedium, a rating of the medium, and/or fees, if any, required to viewthe medium. A person having ordinary skill in the art will appreciatethat many other types of information may be useful for selecting mediain which the user is interested. A search may be performed eitherclient-side by the client viewer module or server-side by the VM inresponse to a request from the client viewer module. The searching maybe performed by comparing inputted or selected search fields to themetadata associated with the media. In a server-side embodiment, theresults of the search can be sent from the VM to the client viewermodule. Results of the search can be displayed by the client viewermodule on a map, in a list, or by some other graphical format. In anembodiment, the search may comprise the user defining a geographicalregion, for example, using a map display, and the results may comprisedisplaying indications of locations of available media on the mapdisplay. In an embodiment where permissions are used, media which theuser does not have permission to view are not displayed in the results.Sponsored media, such as by an advertiser on the Augmented RealityPlatform, may also be displayed. The sponsored media may bedifferentiated from media associated with a user's contacts, such asfriends or members of the user's social groups.

The client viewer module may display, or work in conjunction with anapplication which displays, a map containing the locations associatedwith one or more media available for viewing by the user through theAugmented Reality Platform. The locations may be indicated by icons orthumbnails of the associated media, or by some other means. In anembodiment which utilizes thumbnails, the thumbnails may compriselow-resolution representations of the associated media.

Additionally or alternatively, the client viewer module may display alist containing locations which are associated with available media, alist of the media themselves, a list of the originators of the media, orother representations of the media. In one embodiment, the list is inthe form of a carousel, such that the user may graphically rotatethrough the entries in the list.

In an embodiment, the client viewer module, either automatically or inresponse to a user interaction, notifies the user of the client deviceof nearby locations associated with available media. These locations canbe retrieved by the client viewer module from the VM, for example, whenthe client device is within a predetermined range of the locations. Thenotification may include a message, a sound, a vibration of the clientdevice, the initiation of an application, or the like.

In an embodiment, when the client device is within a range of anavailable medium, the client viewer module automatically opens anAugmented Reality viewer. The Augmented Reality viewer can be a mapdisplay or a view of reality, with indications of available mediasuperimposed over the map display or view of reality. The indicationsmay include, for example, icons, descriptions, low-resolution images orthumbnails of the media, or any other indication which conveys theavailability of an augmented reality opportunity. The indications may besized or otherwise ordered in relation to their distance from the clientdevice. For example, locations of media which are farther from theclient device may have smaller indications than those which are closerto the client device.

According to an embodiment, information can be displayed to enable theuser to determine a value of or interest in the media. This informationcan be displayed in conjunction with the indications (e.g., icons,thumbnails, list entries) or in response to a user interaction, such asselecting the indication. Such information may include, for example, theidentity of the originator or other user associated with the medium,date and/or time the medium was captured, distance to the medium fromthe current location of the client device or another location, type ofmedium, a rating of the medium, descriptions, comments, hyperlinks torelated information, historical information, and/or fees, if any,required to view the medium. For example, the locations of media can berepresented or listed in conjunction with a title, a rating, adescription, an identity of the originator, and the like. A personhaving ordinary skill in the art will appreciate that many other typesof information may be useful for display.

In an embodiment, the user of the client device can also search, sort,or filter the display or list of media based on one or more fields.These fields may include, for example, the identity of the originator orother user associated with the medium, date and/or time the medium wascaptured, distance to the medium from the current location of the clientdevice or another location, type of medium, a rating of the medium,and/or fees, if any, required to view the medium. A person havingordinary skill in the art will appreciate that many other types ofinformation may be useful for selecting media in which the user isinterested. The searching, sorting, and/or filtering may be performedeither client-side by the client viewer module or server-side by the VMin response to a request from the client viewer module. In a server-sideembodiment, the results of the searching, sorting, and/or filtering maybe sent from the VM to the client viewer module.

According to an embodiment, the client viewer module also displayslocations of other points of interest, such as attractions andbusinesses. The client viewer module may also provide routing guidanceto those locations, as well as to locations of media available forviewing. If a user of the client device is following a route from onelocation to another, the client viewer module may display the locationsof available or sponsored media along or near the route. The clientviewer module may also provide a guided route between two or morelocations of media available for augmented viewing.

In an embodiment, when a user of the client device is within apredetermined range of a location associated with a medium available tothe user through the Augmented Reality Platform, the marker and/ormedium is automatically retrieved by the client viewer module from theVM and stored in a memory of the client device. In addition oralternatively, the marker and/or medium can be retrieved and stored in amemory of the client device in response to a user interaction, such as aselection of an icon, thumbnail, or list entry representing the medium.Sensor data and other information associated with the marker and/ormedium, including information concerning the relationship between themarker and medium, can also be retrieved from the VM eithersimultaneously, contemporaneously, or subsequently. In an embodimentwhere the sensor data and other information is retrieved simultaneouslywith the marker and/or medium, the sensor data and other information canbe, partially or wholly, packaged or encoded with the marker and/ormedium, for example, in an EXIFF file.

The user of the client device, may position the client device, such thata view of reality associated with the medium is visible in the opticalviewer. The client viewer module may aid the user in positioning theclient device. For instance, the client viewer module may use the sensordata associated with the medium to provide instructions to the user onhow to position the client device, for example, through verbal cues orby displaying instructions on the client device, including on theoptical viewer. In an embodiment, the client viewer module compares theretrieved sensor data associated with the medium with sensor dataprovided by the client device, and instructs the user how to positionthe client device such that the sensor data provided by the clientdevice matches the retrieved sensor data associated with the medium. Inan embodiment, cues can be given to the user by the client viewer modulethrough the display of the client device. The cues may be overlaid overa view of reality being displayed on the display of the client device.The cues may include, for example, a silhouette, outline, or mosaic ofthe marker, position in which the user should stand, or position ororientation in which the device should be placed. The user may utilizethese cues to properly position himself or herself and properly alignthe marker with the view of reality. In some embodiment, when the useris in the best or near-best alignment, the current view of reality canbe used as the marker, instead of the marker retrieved from the VM. Inthis manner, the old marker is switched out for a more relevant andcurrent marker.

The client viewer module uses the marker for pattern-matching with theview of reality by comparing the marker with a view of reality visiblein the optical viewer. In an embodiment, the client viewer moduleattempts to match the marker or patterns in the marker to the view ofreality or patterns in the view of reality. For example, if both themarker and the view of reality contain a mountain range, the clientviewer module may attempt to align the mountain range in the marker withthe mountain range in the view of reality. A variety of methods andalgorithms may be used to perform the pattern-matching. For example,Metaio™ produces a software development kit (SDK) which includessuitable pattern-matching algorithms. Other entities and products, suchas Eyedea™, Layar™, Qualcomm's Vuforia™, and/or Aurasma™, also providesuitable pattern-matching algorithms. The algorithms can also becombined with proprietary object-tracking algorithms. For instance,based on known patterns, features, or objects in the current view ofreality, the X-Y on-screen locations of the elements can be known,predicted, or determined from frame to frame. In this manner, theon-screen locations of the elements can be traced and tracked based onthe associated pixel orientations (or mathematical/visual derivativesthereof) determined by the matching of the elements in the current viewof reality to the elements in the marker.

In an embodiment, the client viewer module may use the sensor data toaid in pattern-matching. For example, the client viewer module may useinformation concerning the location and orientation of the client deviceat the time that the marker was captured to determine which portions ofthe view of reality should be compared to the marker. An algorithm toweigh each of the available metrics from the sensor data and/or opticalviewer can be utilized to weight the probability that the user is in theproper location and pose/orientation, thereby displaying the propermedia and enhancing the user experience.

Once the client viewer module has aligned the marker with the view ofreality, it can superimpose the medium, which may comprise an effect,over the view of reality. In an embodiment, the client viewer moduleuses information concerning the relationship between the medium andmarker to align the superimposed medium with the view of reality. In anembodiment, if the client viewer module is unable to match the markerwith the view of reality, it may use the location and orientation datacaptured at the time that the marker was captured to align the mediumwith the view of reality. Once the medium has been superimposed over theview of reality, as visible through an optical viewer, the view ofreality has been augmented. In an embodiment, the client viewer modulemay enable the user to increase or decrease the transparency of thesuperimposed medium.

In an embodiment, the client viewer module can utilize apattern-matching algorithm which creates local affine frames andmaximally stable extremal regions within the current view of reality.These frames and regions can then be matched to the previously createdmarker. When comparing the marker to the current view of reality, theclient viewer module can determine a probability or probabilities thatthe user is at or near the same location and at the sameorientation/pose as when the medium was pinned. For instance, aprobability can be measured by comparing the metadata of the marker(e.g., yaw, pitch, roll, direction coordinates, pattern-matching,lighting, etc.) to the current view of reality and/or real-time sensordata from the client device. A match can be confirmed when theprobability meets a certain, predetermined threshold. In an embodiment,once the match is confirmed, the newly created frames and regions can beutilized as the marker that the client viewer module tracks and which isused to augment the view of reality with the pinned digital effect. Inthis manner, the marker is “switched.” The previously generated markeronly serves as a confirmation, and the marker that is actually utilizedby the module is from the current view of reality. This allows the userto enjoy a more relevant experience according to the current localizedlighting and environmental conditions which affect the pattern-matching.In an embodiment, the current lighting and time-of-day conditions can bemeasured by the client device to accommodate and match the best markerfor the pattern comparison and/or marker switching.

Any marker in a memory of the client device or server and available tothe client viewer module can be utilized in the above manner. Forinstance, multiple marker files can be retrieved (e.g., using a query)or generated for a view of reality or a portion or portions of the viewof reality. These multiple marker files can be used to augment the sameview of reality with a plurality of media or effects.

In an embodiment, the client device may comprise one or more secondarycameras or optical viewers. These secondary cameras may be configured toscan a periphery of the client device and conduct markermatching/switching prior to a view of reality coming within view of theprimary camera of the client device. Then, when the client device ismoved, such that the view of reality does come within the view of theprimary camera, the effects can be quickly engaged without the delayrequired for recognition and downloading/uploading of data. In thismanner, multiple markers and their associated media can be readied forrevelation without relying on a single camera or optical viewer, andperformance time, as well as the user experience, can be improved.

In an embodiment, simultaneous location and mapping (SLAM) can beutilized in a multi-camera mode to continually scan and assign theexisting periphery continuation of a three-dimensional medium. Whereas,in a single camera unit, the monocle will only be utilized for theinitial rendering and display of the immediate view of the medium,secondary and tertiary camera units can serve as the seeking andassignment for the continual propagation (i.e., hanging) of the mediumin a continuous and fluid manner. This gives the medium the ability toremain in context as the scene changes with changes in point of view.

In an embodiment, once a user has properly positioned the client device,such that a marker for a particular medium has been matched, additionalmedia and effects may be displayed to the user through the client viewermodule based on a vector position which is relative to the position ofthe matched marker.

In an embodiment, the user may capture a photograph or video of theaugmented view of reality which includes the superimposed medium, usingthe client viewer module or another application which is configured tocommunicate with the optical viewer and the client viewer module. Theaugmented view of reality, as captured in the photograph or video, mayinclude the user or other individuals or objects interacting withreality. In this manner, individuals can “jump” into the augmented viewof reality, and appear to be present in or interacting with theaugmented view of reality, including the superimposed medium. In thescenario where the medium is a historical image, a generational imagecan be obtained which would otherwise be impossible. For example, theuser can “jump” into the augmented (i.e., historical) view of reality,and appear to have traveled back in time.

3. Example Embodiments

Illustrative, non-limiting embodiments of a client application for anAugmented Reality Platform will now be described with reference to thefigures.

a. Client Perspective Module

FIG. 1 illustrates a display for obtaining information about the user ofthe client perspective module, which can be stored in the VM and aid theclient viewer module or user of the client viewer module in aligning amarker and/or medium with a view of reality. Such information mayinclude, for example, whether the user of the client perspective moduleis left-handed or right-handed, and the height of the user. It mayfrequently be the case that the user of the client viewer module withrespect to a particular medium is different-handed, a different height,or different in some other way than the user of the client perspectivemodule who originated that medium. In such cases, the different usersmay have different perspectives of the same view of reality. Thus, itmay be beneficial to have such information about the originator in orderfor the client viewer module to better instruct or otherwise aid adifferent user in orienting the client device so that a marker can bematched with a view of reality and a medium superimposed on the view ofreality.

FIG. 2 illustrates a display for obtaining from the user authenticationinformation required to access the Augmented Reality Platform. Requiredinputs may include a username or email address and a password. In orderto access the Augmented Reality Platform, the user may have the option,or alternatively be required, to authenticate using an account with athird-party service provider, such as Facebook®. The display may alsoinclude an option to register for an account, either with the AugmentedReality Platform or a third-party service provider.

FIG. 3 illustrates a display by which a user may view and select amedium to “hang,” i.e., superimpose over a view of reality. The clientperspective module may comprise a wizard which walks a user through thesteps of “hanging” or “pinning” a medium to a view of reality. Thewizard may comprise one or more screens, which guide the user throughthe hanging process. It should be understood that one or more of thewizard screens may be skipped by the user and/or subsequently returnedto by the user. The user may select the medium 310, in this case animage, from a list of media 320 stored in a memory on a client device.For example, the user may interact with a touch-screen of the clientdevice by tapping on the medium 310 or dragging the medium 310 from thelist 320 to the selected medium region 330. Once the user has selectedthe medium or media which the user wishes to hang, the user can interactwith the user interface, for example, by pressing an action button 340to move to the next screen of the wizard.

FIG. 4 illustrates a display by which the user may input information tobe associated with the selected medium or media. This display may beshown, for example, automatically after a user has selected the medium,automatically after a user has hung a selected medium, or at a differenttime in response to a user interaction with the client perspectivemodule. The inputs may comprise textboxes into which the user can entera title 410, location description 420, date that the medium was captured440, and/or a description of the medium 450. The inputs may also includea GPS reference 430. The GPS reference input may comprise a map display,as shown in FIG. 5, which enables the user to indicate a location of themedium on the map, for example, by tapping (e.g., via a touch-screeninterface of the client device) a point on the map. This interaction mayresult in a pin being dropped on the tapped location. The GPS referenceinput can be used as location information associated with the medium andmay be in addition to or instead of data from a GPS sensor of the clientdevice. A person having ordinary skill in the art will appreciate thatit may beneficial to obtain additional information as well. Some of theinformation may be pre-populated based on information associated withthe image file (e.g., metadata) or information acquired from the clientdevice (e.g., GPS coordinates for GPS reference 430).

FIG. 6 illustrates a display by which a user can view and/or selectmedia stored in a memory on the client device. The media may becategorized, for example, according to their progress within the clientperspective module. For instance, one category 610 can represent mediafor which all required information from the user has been providedand/or the location of the media is within a range of the client device.A second category 620 can represent media for which all requiredinformation from the user has been provided but the location of themedia is not within a range of the client device. A third category 630can represent media for which not all required information has beenprovided. Further categories or a different combination of categoriesmay also be provided. In response to a user interaction, such asselecting a medium by, for example, tapping an icon or thumbnailassociated with the medium, the client perspective module may generatean input display for entering information, such as shown in FIG. 7. Theinput display may be similar or identical to the input display in FIG.4, or different from the input display in FIGS. 4 and 7.

Once the user has indicated a selection of a medium to hang, the clientperspective module may display a view of reality, as shown in FIG. 8,along with the medium superimposed transparently over the view ofreality, as shown in FIG. 9. The user may then align the transparentsuperimposed medium with the view of reality, as shown in FIG. 10, forexample, by manipulating (e.g., resizing or rotating) the superimposedmedium, reorienting the client device, or both. In an embodiment, thismanipulation is performed by interacting with a touch-screen of theclient device (e.g., pinching fingers together to shrink a medium,spreading fingers apart to enlarge a medium, rotating two fingers torotate a medium, etc.).

FIG. 11 illustrates an overlay of the view of reality, as seen throughthe optical viewer. The overlay can aid the user in aligning thesuperimposed medium with the view of reality or provide the user withinformation about the view of reality. For example, the overlay mayinclude location information, such as an altitude 1120, GPS coordinates1130, and/or a heading 1140. The overlay may also include an actionbutton 1110, such that when the user selects the action button 1110, thecurrent view of reality in the optical viewer or a portion of the viewof reality is captured as a marker. The overlay can appear differentlybased on the handedness of the user, i.e., whether the user isleft-handed or right-handed. For example, the positions of informationand interactive buttons or other features of the overlay can bedifferent (e.g., on different sides of the display) based on thehandedness of the user.

Selection of the action button 1110 on the overlay may result in a timer1150 being displayed, as shown in FIG. 12. The timer 1150 may be set fora predetermined duration, and may comprise a partially transparentoverlay. The user may continue to adjust the superimposed medium and/orthe client device for the duration of the timer 1150. Once the timer1150 expires, the client perspective module can capture the marker.Alternatively, the user may be required to hold the client device steadyfor the duration of the timer 1150, permitting the client perspectivemodule to capture information regarding the augmented view of realityfor the duration of the timer 1150.

According to an embodiment, since the client device is physically at thelocation at which the medium was captured, sensor data can besimultaneously obtained and measured as it actually occurs in the field.A system which uses images of the location (e.g., images collected fromGoogle® Street View) or second-hand sensor data to position the mediummay not be as accurate, since it can not account for interferences(e.g., magnetic, electrical, etc.) which might affect the client deviceat the physical location and having the orientation at which the mediumis to be viewed. By enabling users to physically take the client deviceto the location at which the medium is to be viewed and to capture themarker and sensor data simultaneously, more accurate sensor data andpositioning information may be obtained. This can aid in the use of theclient viewer module, as discussed in detail below. For instance, themore accurate sensor data can aid the user of the client viewer modulein positioning the client device, and/or aid the client viewer module insuperimposing the medium over a view of reality.

In an embodiment, if the client perspective module is not able tosufficiently capture the marker, its relationship to the medium, and/orthe required sensor data, the user may be prompted to retry the captureprocess. Once the marker has been captured successfully, the user can beprompted to confirm that the capture is acceptable, as shown in FIG. 13.If the user confirms that the capture was acceptable, the user may beprompted to upload the captured information to the VM, or to performadditional captures for other media stored in the memory of the clientdevice, as shown in FIG. 14. If the user elects to upload the capturedinformation, the captured information, including the marker, medium,sensor data, and/or other information, may be packaged into a save ormodify request and sent to the VM. Sensor data and other informationassociated with the media can be, partially or wholly, packaged orencoded with the associated medium and/or marker, for example, in anEXIFF file, an example representation of which is shown in FIG. 15.Alternatively, the captured information may be packaged and sent to theVM automatically after successful capture or once the user confirms thatthe capture was acceptable.

The VM is configured to receive the save or modify request and/or thepackaged captured information, and store the information in one or moredatabases for future retrieval by the client viewer module ormodification by the client perspective module. FIG. 26 illustrates ahigh-level diagram of a process for storing and retrieving media,according to an embodiment. In step 2610, the VM receives a medium,marker, and/or metadata associated with the medium and/or marker, forexample, from a client perspective module executing on a client device.In step 2620, the medium, marker, and/or metadata are stored in one ormore databases either locally or remotely accessible to the VM.Subsequently, in step 2630, the VM may receive a request for theparticular medium, for example, from a client viewer module executing ona client device. In step 2640, in response to the request, the VMretrieves the medium, marker, and/or metadata from the one or moredatabases and send the retrieved data to the client viewer module of theclient device.

FIG. 27 illustrates a high-level diagram of a process for generating amarker, according to an embodiment of the client perspective module. Instep 2710, the medium is aligned with a view of reality. This alignmentmay be performed by the client perspective module and, in someembodiments, aided by the user of the client device executing the clientperspective module. In step 2720, the marker is generated based on theview of reality and the alignment. The marker is then uploaded to the VMin step 2740.

b. Client Viewer Module

FIG. 16 illustrates a display for obtaining information about the userof the client viewer module, which can aid the client viewer module oruser of the client viewer module in aligning a marker and/or medium witha view of reality. Such information may include, for example, whetherthe user of the client viewer module is left-handed or right-handed, andthe height of the user. It may frequently be the case that the user ofthe client viewer module with respect to a particular medium isdifferent-handed, a different height, or different in some other waythan the user of the client perspective module who originated thatmedium. In such cases, the different users may have differentperspectives of the same view of reality. Thus, it may be beneficial tohave such information about the user of the client viewer module inorder for the client viewer module to better instruct or otherwise aidthe user in orienting the client device so that a marker can be matchedwith a view of reality and a medium superimposed on the view of reality.

FIG. 17 illustrates a display for obtaining from the user authenticationinformation, which in an embodiment is required to access the AugmentedReality Platform. Required inputs may include a username or emailaddress and a password. In order to access the Augmented RealityPlatform, the user may have the option, or alternatively be required, toauthenticate using an account with a third-party service provider, suchas Facebook®. The display may also include an option to register for anaccount, either with the Augmented Reality Platform or a third-partyservice provider.

FIG. 18 illustrates a display for listing media available for viewing bya user of the client viewer module. The display may list available mediaat nearby locations or locations within a range of the client device,including a thumbnail of the medium and/or a representation 1810 of theoriginator of the medium, the originator 1820 of the medium, the date orapproximate date 1830 of capture, the number of times the medium hasbeen viewed 1840, the distance 1850 of the medium from the clientdevice, and the like. The display may list the media by originator ofthe media, including the number of media available 1860 from thatoriginator at a particular location, and in some embodiments, a numberof media recently made available 1870 from the originator at theparticular location.

The display may include filter options 1880, for example, to enable theuser to limit the listing of media to media available at locationswithin a certain distance or radius of the client device. The displaymay also include sort options 1890, to enable the user to sort availablemedia based on, for example, date that the medium was captured,originator of the medium, whether or not the medium has been previouslyviewed by the user, ratings, most viewed media, and/or distance. Thedisplay may also include a listing of media which are sponsored by anorganization, for example, by an advertiser on the Augmented RealityPlatform. Sponsored media can be distinguished from non-sponsored media,so that it is clear to the user which media are sponsored by advertisersand which are not.

FIG. 19 illustrates a display for listing available media using acarousel format. The carousel format can be displayed when the clientdevice is within a predetermined or set range of the media. In anembodiment, the carousel displays media which are available from asingle originator at or near the location of the client device. Thecarousel enables the user to rotate entries for media between aforeground 1910 and background 1920 and displays information concerningthe media, such as a title, originator, distance, rating, and whether ornot the user of the client device has previously viewed the media.

FIG. 20 illustrates a display for listing available media using a mapdisplay. The locations of available media may be indicated on the mapdisplay using icons, such as pushpins. In an embodiment, the mapdisplays media which are available from a single originator at or nearthe location of the client device. The icons may be interactive. Forexample, a selection of the icons by the user may result in displays ofadditional information related to the associated media.

FIG. 21 illustrates an Augmented Reality Window. The Augmented RealityWindow can be automatically displayed when the user gets close to anavailable medium or media. The client device may chime, vibrate, orotherwise alert the user of the client device that an augmented realityopportunity is available. This alert or notification can be performed inconjunction with the display of the Augmented Reality Window. As shown,the Augmented Reality Window can display a representation 2110 of eachmedium (e.g., icon, title, description, etc.) at a position in thewindow which corresponds to the position of the corresponding medium.

The client viewer module may retrieve information concerning theavailable media from the VM, including the marker, medium, informationconcerning the relationship of the marker and medium, sensor data,and/or other information. This information may be packaged or encodedtogether, for example, in an EXIFF file. The client viewer module mayautomatically retrieve the media information when the client device iswithin a predetermined or set range of the location associated with themedia, or may retrieve the media in response to a user interaction, suchas a selection of the media for viewing.

In an embodiment, the Augmented Reality Window is in communication witha camera feature or other optical viewer of the client device, anddisplays a view of reality with superimposed indications 2210 ofavailable media, such as icons, descriptions, or the media themselves.As shown in FIG. 22, the Augmented Reality Window can also displayinstructions 2220 to aid the user of the client device in positioningthe client device. For example, an instruction such a “turn left,” “turnright,” “up,” “down,” and the like may be displayed. A target 2230 orarrows, indicating where the user should stand or position the clientdevice can also be displayed. The instructions 2220 may be superimposedover the view of reality.

In an embodiment, as the user is positioning the client device, theclient viewer module attempts to pattern-match the marker to a view ofreality in the Augmented Reality Window. The client viewer module mayuse the sensor data to aid in aligning the marker with the view ofreality. Once the client viewer module has determined where the markermatches the view of reality, the client viewer module can superimposethe medium 2310 over the view of reality 2320 using the capturedinformation concerning the relationship of the medium to the marker, asshown in FIG. 23. A sponsored message 2410 may also be displayed before,after, or simultaneously with the superimposition of the medium, asshown in FIG. 24. Additional information 2510 associated with thesuperimposed medium can also be displayed, as shown in FIG. 25. Theadditional information 2510 can be displayed, for example, in responseto a user interaction, such as the selection of an information button2330 as shown in FIG. 23.

In an embodiment, the client viewer module can superimpose a video overa view of reality. In this embodiment, the client perspective module mayobtain real-world markers for each frame or a subset of frames from thevideo. The real-world markers may comprise the frames of the videomedium itself. The client viewer module may then performpattern-matching between the real-world markers and the view of realityin order to superimpose frames of the video medium onto the view ofreality. Where a video was not taken from a stationary position, i.e.,the camera position changes from frame to frame, the superimposition ofthe medium may also change such that it appears to move within theaugmented view of reality. In such an embodiment, the pixel locations ofthe superimposed medium are adjusted as the background of the mediumcontinually changes. For example, the medium may be a video of anindividual skateboarding from left to right, with the camera view usedto capture the medium following the skateboarder as he moves. In theevent that the client device is held stationary during viewing, thesuperimposed medium will appear to move from left to right along withthe skateboarder within the augmented view of reality. However, if theclient device is moved such that the view of reality stays in alignmentwith the camera views used to capture the medium, then the superimposedmedium will appear to remain stationary within the augmented view ofreality.

Once the medium has been superimposed over the view of reality in theAugmented Reality Window, a user can capture images or videos of theaugmented reality using a camera feature or other optical viewer of theclient device. Images or videos of the augmented reality can be capturedby the user in the same manner as the user would capture an image orvideo of reality. The client viewer module or a separate module isconfigured to capture the view of reality and superimpose the mediumover the view of reality to create a single image or video of anaugmented reality. In this manner, the user or other individuals maystep into the view of reality as seen through the Augmented RealityWindow, and the resulting image or video will contain the view ofreality, the user or other individuals, and the superimposed mediumtogether. In the case where the superimposed medium is historical, theuser or other individuals may appear to have traveled back in time. Inthe case where the superimposed medium is a future design or depiction,the user or other individuals may appear to have traveled forward intime. In this way, the Augmented Reality Platform is capable ofdelivering a truly unique experience for each and every user.

FIG. 28 illustrated a high-level block diagram of a process foraugmenting a view of reality, according to an embodiment of the clientviewer module. In step 2810, the desired medium, along with its markerand/or metadata is retrieved from the VM by the client viewer moduleexecuted on a client device. In step 2820, the marker is matched to acurrent view of reality of the client device, for example, usingpattern-matching. Once the marker is matched to the view of reality, instep 2830, the medium is overlaid or superimposed over the view ofreality based on the relationship of the medium and the marker. Thisrelationship may be stored as at least a portion of the metadata. Instep 2840, once the medium is overlaid over the view of reality, theaugmented view of reality is displayed to the user of the client viewermodule.

5. Additional Features

In an embodiment, access to the VM can be restricted such that it isonly accessible through the client application. In this embodiment,there is no user interface or interaction with the VM, except via theclient application or applications. For additional security,communications between the VM and client application can be encrypted,for example, using Transport Layer Security (TLS) or Secure SocketsLayer (SSL).

In an embodiment, media can be filtered to prevent or restrict uploadingor viewing of indecent media. The filtering process may be performed bythe VM or client application. For example the VM or client applicationmay be configured to identify indecent photographs throughpattern-matching or other algorithms. Additionally or alternatively,filtering can be performed by users of the Augmented Reality Platform.For example, the client application or VM may enable a user to flagmedia which the user believes is indecent. A flag value indicatingwhether the medium is indecent or potentially indecent can be associatedwith the flagged medium and stored in the VM. The flag may be a Booleanvalue.

The VM or client application can prevent media which has been identifiedas indecent from being stored in the VM. Additionally or alternatively,media which has been flagged as indecent or potentially indecent can bestored and subsequently reviewed to determine whether it should beremoved from the VM. Alternatively, viewing of indecent media can berestricted to users with certain permissions and/or characteristics,e.g., users over the age of eighteen.

FIG. 29 is a block diagram illustrating an example wired or wirelesssystem 550 that may be used in connection with various embodimentsdescribed herein. For example the system 550 may be used as or inconjunction with one or more of the mechanisms and processes previouslydescribed above. The system 550 can be a server or any conventionalpersonal computer, or any other processor-enabled device that is capableof wired or wireless data communication. Other computer systems and/orarchitectures may be also used, as will be clear to those skilled in theart.

The system 550 preferably includes one or more processors, such asprocessor 560. Additional processors may be provided, such as anauxiliary processor to manage input/output, an auxiliary processor toperform floating point mathematical operations, a special-purposemicroprocessor having an architecture suitable for fast execution ofsignal processing algorithms (e.g., digital signal processor), a slaveprocessor subordinate to the main processing system (e.g., back-endprocessor), an additional microprocessor or controller for dual ormultiple processor systems, or a coprocessor. Such auxiliary processorsmay be discrete processors or may be integrated with the processor 560.

The processor 560 is preferably connected to a communication bus 555.The communication bus 555 may include a data channel for facilitatinginformation transfer between storage and other peripheral components ofthe system 550. The communication bus 555 further may provide a set ofsignals used for communication with the processor 560, including a databus, address bus, and control bus (not shown). The communication bus 555may comprise any standard or non-standard bus architecture such as, forexample, bus architectures compliant with industry standard architecture(“ISA”), extended industry standard architecture (“EISA”), Micro ChannelArchitecture (“MCA”), peripheral component interconnect (“PCI”) localbus, or standards promulgated by the Institute of Electrical andElectronics Engineers (“IEEE”) including IEEE 488 general-purposeinterface bus (“GPIB”), IEEE 696/S-100, and the like.

System 550 preferably includes a main memory 565 and may also include asecondary memory 570. The main memory 565 provides storage ofinstructions and data for programs executing on the processor 560, suchas one or more of the modules discussed above. The main memory 565 istypically semiconductor-based memory such as dynamic random accessmemory (“DRAM”) and/or static random access memory (“SRAM”). Othersemiconductor-based memory types include, for example, synchronousdynamic random access memory (“SDRAM”), Rambus dynamic random accessmemory (“RDRAM”), ferroelectric random access memory (“FRAM”), and thelike, including read only memory (“ROM”).

The secondary memory 570 may optionally include a internal memory 575and/or a removable medium 580, for example a floppy disk drive, amagnetic tape drive, a compact disc (“CD”) drive, a digital versatiledisc (“DVD”) drive, etc. The removable medium 580 is read from and/orwritten to in a well-known manner. Removable storage medium 580 may be,for example, a floppy disk, magnetic tape, CD, DVD, SD card, etc.

The removable storage medium 580 is a non-transitory computer readablemedium having stored thereon computer executable code (i.e., software)and/or data. The computer software or data stored on the removablestorage medium 580 is read into the system 550 for execution by theprocessor 560.

In alternative embodiments, secondary memory 570 may include othersimilar means for allowing computer programs or other data orinstructions to be loaded into the system 550. Such means may include,for example, an external storage medium 595 and an interface 570.Examples of external storage medium 595 may include an external harddisk drive or an external optical drive, or and external magneto-opticaldrive.

Other examples of secondary memory 570 may include semiconductor-basedmemory such as programmable read-only memory (“PROM”), erasableprogrammable read-only memory (“EPROM”), electrically erasable read-onlymemory (“EEPROM”), or flash memory (block oriented memory similar toEEPROM). Also included are any other removable storage media 580 andcommunication interface 590, which allow software and data to betransferred from an external medium 595 to the system 550.

System 550 may also include a communication interface 590. Thecommunication interface 590 allows software and data to be transferredbetween system 550 and external devices (e.g. printers), networks, orinformation sources. For example, computer software or executable codemay be transferred to system 550 from a network server via communicationinterface 590. Examples of communication interface 590 include a modem,a network interface card (“NIC”), a wireless data card, a communicationsport, a PCMCIA slot and card, an infrared interface, and an IEEE 1394fire-wire, just to name a few.

Communication interface 590 preferably implements industry promulgatedprotocol standards, such as Ethernet IEEE 802 standards, Fiber Channel,digital subscriber line (“DSL”), asynchronous digital subscriber line(“ADSL”), frame relay, asynchronous transfer mode (“ATM”), integrateddigital services network (“ISDN”), personal communications services(“PCS”), transmission control protocol/Internet protocol (“TCP/IP”),serial line Internet protocol/point to point protocol (“SLIP/PPP”), andso on, but may also implement customized or non-standard interfaceprotocols as well.

Software and data transferred via communication interface 590 aregenerally in the form of electrical communication signals 605. Thesesignals 605 are preferably provided to communication interface 590 via acommunication channel 600. In one embodiment, the communication channel600 may be a wired or wireless network, or any variety of othercommunication links. Communication channel 600 carries signals 605 andcan be implemented using a variety of wired or wireless communicationmeans including wire or cable, fiber optics, conventional phone line,cellular phone link, wireless data communication link, radio frequency(“RF”) link, or infrared link, just to name a few.

Computer executable code (i.e., computer programs or software) is storedin the main memory 565 and/or the secondary memory 570. Computerprograms can also be received via communication interface 590 and storedin the main memory 565 and/or the secondary memory 570. Such computerprograms, when executed, enable the system 550 to perform the variousfunctions of the present invention as previously described.

In this description, the term “computer readable medium” is used torefer to any non-transitory computer readable storage media used toprovide computer executable code (e.g., software and computer programs)to the system 550. Examples of these media include main memory 565,secondary memory 570 (including internal memory 575, removable medium580, and external storage medium 595), and any peripheral devicecommunicatively coupled with communication interface 590 (including anetwork information server or other network device). Thesenon-transitory computer readable mediums are means for providingexecutable code, programming instructions, and software to the system550.

In an embodiment that is implemented using software, the software may bestored on a computer readable medium and loaded into the system 550 byway of removable medium 580, I/O interface 585, or communicationinterface 590. In such an embodiment, the software is loaded into thesystem 550 in the form of electrical communication signals 605. Thesoftware, when executed by the processor 560, preferably causes theprocessor 560 to perform the inventive features and functions previouslydescribed herein.

The system 550 also includes optional wireless communication componentsthat facilitate wireless communication over a voice and over a datanetwork. The wireless communication components comprise an antennasystem 610, a radio system 615 and a baseband system 620. In the system550, radio frequency (“RF”) signals are transmitted and received overthe air by the antenna system 610 under the management of the radiosystem 615.

In one embodiment, the antenna system 610 may comprise one or moreantennae and one or more multiplexors (not shown) that perform aswitching function to provide the antenna system 610 with transmit andreceive signal paths. In the receive path, received RF signals can becoupled from a multiplexor to a low noise amplifier (not shown) thatamplifies the received RF signal and sends the amplified signal to theradio system 615.

In alternative embodiments, the radio system 615 may comprise one ormore radios that are configured to communicate over various frequencies.In one embodiment, the radio system 615 may combine a demodulator (notshown) and modulator (not shown) in one integrated circuit (“IC”). Thedemodulator and modulator can also be separate components. In theincoming path, the demodulator strips away the RF carrier signal leavinga baseband receive audio signal, which is sent from the radio system 615to the baseband system 620.

If the received signal contains audio information, then baseband system620 decodes the signal and converts it to an analog signal. Then thesignal is amplified and sent to a speaker. The baseband system 620 alsoreceives analog audio signals from a microphone. These analog audiosignals are converted to digital signals and encoded by the basebandsystem 620. The baseband system 620 also codes the digital signals fortransmission and generates a baseband transmit audio signal that isrouted to the modulator portion of the radio system 615. The modulatormixes the baseband transmit audio signal with an RF carrier signalgenerating an RF transmit signal that is routed to the antenna systemand may pass through a power amplifier (not shown). The power amplifieramplifies the RF transmit signal and routes it to the antenna system 610where the signal is switched to the antenna port for transmission.

The baseband system 620 is also communicatively coupled with theprocessor 560. The central processing unit 560 has access to datastorage areas 565 and 570. The central processing unit 560 is preferablyconfigured to execute instructions (i.e., computer programs or software)that can be stored in the memory 565 or the secondary memory 570.Computer programs can also be received from the baseband processor 610and stored in the data storage area 565 or in secondary memory 570, orexecuted upon receipt. Such computer programs, when executed, enable thesystem 550 to perform the various functions of the present invention aspreviously described. For example, data storage areas 565 may includevarious software modules (not shown).

Various embodiments may also be implemented primarily in hardware using,for example, components such as application specific integrated circuits(“ASICs”), or field programmable gate arrays (“FPGAs”). Implementationof a hardware state machine capable of performing the functionsdescribed herein will also be apparent to those skilled in the relevantart. Various embodiments may also be implemented using a combination ofboth hardware and software.

Furthermore, those of skill in the art will appreciate that the variousillustrative logical blocks, modules, circuits, and method stepsdescribed in connection with the above described figures and theembodiments disclosed herein can often be implemented as electronichardware, computer software, or combinations of both. To clearlyillustrate this interchangeability of hardware and software, variousillustrative components, blocks, modules, circuits, and steps have beendescribed above generally in terms of their functionality. Whether suchfunctionality is implemented as hardware or software depends upon theparticular application and design constraints imposed on the overallsystem. Skilled persons can implement the described functionality invarying ways for each particular application, but such implementationdecisions should not be interpreted as causing a departure from thescope of the invention. In addition, the grouping of functions within amodule, block, circuit or step is for ease of description. Specificfunctions or steps can be moved from one module, block or circuit toanother without departing from the invention.

Moreover, the various illustrative logical blocks, modules, and methodsdescribed in connection with the embodiments disclosed herein can beimplemented or performed with a general purpose processor, a digitalsignal processor (“DSP”), an ASIC, FPGA or other programmable logicdevice, discrete gate or transistor logic, discrete hardware components,or any combination thereof designed to perform the functions describedherein. A general-purpose processor can be a microprocessor, but in thealternative, the processor can be any processor, controller,microcontroller, or state machine. A processor can also be implementedas a combination of computing devices, for example, a combination of aDSP and a microprocessor, a plurality of microprocessors, one or moremicroprocessors in conjunction with a DSP core, or any other suchconfiguration.

Additionally, the steps of a method or algorithm described in connectionwith the embodiments disclosed herein can be embodied directly inhardware, in a software module executed by a processor, or in acombination of the two. A software module can reside in RAM memory,flash memory, ROM memory, EPROM memory, EEPROM memory, registers, harddisk, a removable disk, a CD-ROM, or any other form of storage mediumincluding a network storage medium. An exemplary storage medium can becoupled to the processor such the processor can read information from,and write information to, the storage medium. In the alternative, thestorage medium can be integral to the processor. The processor and thestorage medium can also reside in an ASIC.

The above description of the disclosed embodiments is provided to enableany person skilled in the art to make or use the invention. Variousmodifications to these embodiments will be readily apparent to thoseskilled in the art, and the generic principles described herein can beapplied to other embodiments without departing from the spirit or scopeof the invention. Thus, it is to be understood that the description anddrawings presented herein represent a presently preferred embodiment ofthe invention and are therefore representative of the subject matterwhich is broadly contemplated by the present invention. It is furtherunderstood that the scope of the present invention fully encompassesother embodiments that may become obvious to those skilled in the artand that the scope of the present invention is accordingly not limited.

What is claimed is:
 1. A system comprising: at least one hardwareprocessor; and one or more modules configured to, when executed by theat least one hardware processor, receive a medium, a first marker, andmetadata from a depository, wherein the first marker comprises at leasta portion of a view of reality, match the first marker to at least afirst portion of the view of reality, after the first marker is matchedto the at least a first portion of the view of reality, generate asecond marker based on at least a second portion of the view of reality,and upload the second marker to the depository, and superimpose themedium over at least a portion of the view of reality, based on at leastone of the first marker and the second marker, to generate an augmentedview of reality.
 2. The system of claim 1, wherein the metadatacomprises position data, the position data comprising at least one of alocation and orientation of a device that generated the marker at a timewhen the marker was generated.
 3. The system of claim 2, wherein thesystem comprises a client device, and wherein the one or more modulesare further configured to provide instructions to a user of the clientdevice on how to position the client device based on the position data.4. The system of claim 1, wherein the system comprises a client device,wherein the metadata comprises a location of a device that generated themarker at a time when the marker was generated, and wherein the one ormore modules are further configured to automatically retrieve at leastthe medium and the first marker from the depository when the clientdevice is within a predetermined range of the location.
 5. The system ofclaim 1, wherein the metadata comprises a relationship between themedium and the first marker, and wherein the one or more modules arefurther configured to superimpose the medium over the at least a firstportion or the at least a second portion of the view of reality based onthe relationship between the medium and the first marker.
 6. The systemof claim 1, wherein the one or more modules are further configured tomatch the first marker to the at least first portion of the view ofreality by matching one or more patterns in the first marker to one ormore patterns in the view of reality.
 7. The system of claim 2, whereinthe system comprises a client device, and wherein the one or moremodules are further configured to match the first marker to the at leasta first portion of the view of reality by matching one or more patternsin the first marker to one or more patterns in the view of reality andmatching the position data to at least one of a location and orientationof the client device.
 8. The system of claim 7, wherein the one or moremodules are configured to: attempt to match one or more patterns in thefirst marker to one or more patterns in the view of reality; and, whenit cannot match one or more patterns in the first marker to one or morepatterns in the view of reality, attempt to match the position data toat least one of a location and orientation of the client device.
 9. Thesystem of claim 7, wherein the one or more modules are configured toattempt to match the position data to at least one of a location andorientation of the client device by providing instructions to a user ofthe client device, wherein the instructions are based on the positiondata and the at least one of the location and orientation of the clientdevice.
 10. The system of claim 1, wherein the client viewer module isfurther configured to store an image of the augmented view of reality ina memory.
 11. The system of claim 1, wherein the system comprises aclient device, wherein generating the second marker comprises generatinga high-density image using one or more omni-directional cameras, whereinthe second marker comprises a high-density image, and wherein matchingthe second marker to the least a second portion of the view of realitycomprises approximating the relative position between the client deviceand the second marker using inverse photogrammetry.
 12. The system ofclaim 1, further comprising an embeddable code, wherein the embeddablecode, when executed: superimposes the medium over a captured portion ofthe view of reality; and changes a transparency of the superimposedmedium in response to one or more user interactions.
 13. The system ofclaim 1, wherein the at least a first portion of the view of reality andthe at least a second portion of the view of reality are identical. 14.The system of claim 1, wherein the at least a first portion of the viewof reality and the at least a second portion of the view of reality aredifferent.
 15. A method comprising, by a client device comprising atleast one processor and at least one memory: receiving a medium, a firstmarker, and metadata from a depository, wherein the first markercomprises at least a portion of a view of reality; matching the firstmarker to at least a first portion of the view of reality; after thefirst marker is matched to the at least a first portion of the view ofreality, generating a second marker based on at least a second portionof the view of reality, and uploading the second marker to thedepository; and superimposing the medium over at least a portion of theview of reality, based on at least one of the first marker and thesecond marker, to generate an augmented view of reality.
 16. The methodof claim 15, wherein the metadata comprises position data, the positiondata comprising at least one of a location and orientation of a seconddevice at a time when the marker was generated.
 17. The method of claim16, further comprising instructing a user of the client device on how toposition the client device based on the position data.
 18. The method ofclaim 17, wherein the instructions comprise an outline of the marker.19. The method of claim 15, wherein the metadata comprises arelationship between the medium and the marker, and whereinsuperimposing the medium over the at least a first portion or the atleast a second portion of the view of reality is based on therelationship between the medium and the marker.
 20. The method of claim16, wherein the matching comprises: attempting to match one or morepatterns in the first marker to one or more patterns in the view ofreality; and if unable to match one or more patterns in the first markerto one or more patterns in the view of reality, attempting to match theposition data to at least one of a location and orientation of theclient device.
 21. The method of claim 20, wherein attempting to matchthe position data to at least one of a location and orientation of theclient device comprises providing instructions to a user of the clientdevice based on the position data and the at least one of the locationand orientation of the client device.
 22. The method of claim 15,further comprising storing an image of the augmented view of reality ina memory.
 23. A non-transitory computer-readable medium havinginstructions stored thereon, wherein the instructions, when executed bya processor, cause the processor to: receive a medium, a first marker,and metadata from a depository, wherein the first marker comprises atleast a portion of a view of reality; match the first marker to at leasta first portion of the view of reality; after the first marker ismatched to the at least a first portion of the view of reality, generatea second marker based on at least a second portion of the view ofreality, and upload the second marker to the depository; and superimposethe medium over at least a portion of the view of reality, based on atleast one of the first marker and the second marker, to generate anaugmented view of reality.